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Introduction 


As educators, we have no shortage of expert perspectives on the challenge of improving 
educational outcomes in meaningful, scalable, and equitable ways. Some see a challenge 
of reliably scaling promising practice and ask us to learn faster and scale slower when 
we find promising ideas for improvement.’ Others see school structures built for a 
different era and push us to turn schools into true learning organizations. And some 
see reform idea after reform idea fail over and over in the exact same ways and challenge 
us to design strategies for improvement that take into account both the complexity of 
educational systems and the realities of the day-to-day work of educators in schools.” 
These different perspectives all push us to think more carefully about the systems that 
produce our current results and the systems that may produce different, better results. 
They also challenge us to think about the thorny question of how we measure success in 
schools. How do we know what is working and what is not in our schools? How do we 


measure our success over time? 


Chicago provides a particularly valuable case study for thinking about the challenges we 
face as a nation in creating large-scale, equitable improvements in students’ outcomes. 
Supported by research, data, and practice partners from the University of Chicago, the 
question of what should we measure has fueled a shift in schools, in the school district, 
and across the city in how we quantify success. An overarching focus on the educational 
attainment of Chicago’s youth has changed both the kinds of metrics educators pay 


attention to, and the ways those metrics are used in schools. 


In the past decade or so, the Chicago Public Schools (CPS) has seen tremendous 
improvement on the most important indicators for student success. Overall, between 
2006 and 2017, the district has seen a 28 percentage-point rise in the proportion of 
freshman on-track to graduate, with the greatest increases occurring among Black 
and Latino males. High school graduation rates have increased by 18 percentage 
points, with ACT scores improving at the same time. Most impressively, even with 
thousands of more high school graduates, the four-year college enrollment rate rose 
by 14 percentage points, and the overall proportion of high school freshmen who are 
projected to earn a bachelor’s degree has doubled in the last decade." 


There is an important story to be told about the role that data played—and continues 
to play—in these kinds of improvements. Of course, it would be a mistake to think 
that the mere presence of data provided educators with all they needed to chart such 

a remarkable course of improvement. This decade of improvement in Chicago relied 
on many supports and conditions other than data. Not all data efforts during this time 
period were successful. Not all indicators have taken root in the life of schools. Not all 
educators welcomed data-driven improvement efforts with open arms. And not all data 
systems gave educators the tools needed to guide their efforts and solve their problems 


of practice. In short, not all data are created equal. This era did, however, generate 
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a particular kind of approach to using data in schools in Chicago—an approach 

that responds to the drive to make the field of education more data driven, but also 
acknowledges the still-echoing voices of Tony Bryk and Charles Payne that remind 

us to see the complex systems of human beings that produce the results. We call this 
approach “practice-driven data.” This approach has no playbook, no singular practice, 
no such thing as fidelity. Rather, it is rooted in deep respect for professionalism, a 
recognition of context and autonomy, and an acceptance that implementation is often 
as messy and ambiguous as the system itself. 


This paper is designed to share the lessons we have learned about how, when, why, 
and under what conditions we have seen this practice-driven data approach support 
real and sustainable improvement. The paper is our attempt to share these lessons with 
advocates, policymakers, district and school leaders, and school support organizations, 
both locally and nationally, so they can develop the conditions that will support 


effective data use in schools and school systems. These lessons are: 


| Figure 1; Five Lessons of Practice-Driven Data 
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Taken together, these five lessons from Chicago form an approach to data use focusing 
stakeholders at various levels on the most important goals and features of an ecosystem 
of data use that has the potential to catalyze systematic improvement in student 
outcomes. First, it is important to build capacity to facilitate hard conversations and use 
data collectively to spur action, which requires an emphasis on trust, collaboration, and 
culture. Second, the wide availability of data and the scarce resource of time requires 
that educators and leaders prioritize research-based indicators that matter most for 
students’ success. Third, educators can use relevant research evidence to make meaning 
of the data they use, and to develop shared ownership over the implications of the research. 
Fourth, using data effectively to guide practice requires that educators and leaders use 
the right data at the right time of the school year. Finally, there is no more important use 
of data in public schools than as a tool to identify and stop inequities that continue to 


leave the most vulnerable students further and further behind. 


For each of the lessons above, we describe how the idea has played out in practice 

in Chicago high schools, as well as the implications for different stakeholders in the 
broader education system. We begin by discussing the lesson from the perspective 

of school-based educators and the capacities, skills, and tools they need to use data 
effectively to guide their practice. Then, we consider the implications for the data system 
designers who need to create the technical infrastructure of a data system, generate 
effective data displays that reach users of the data, and also need to support a culture 

of data use in order to enable those within and across schools to make the best use of 
the data in practice. Finally, we look at the considerations for researchers and analysts, 
whose work to understand important predictors, relationships, and outcomes guide the 


development and use of data at a high level. 


We write this paper as three entities that are a part of the University of Chicago 

and work with CPS, but are independent of the district. The University of Chicago 
Consortium on School Research (UChicago Consortium) conducts research of high 
technical quality that can inform and assess policy and practice in CPS. The UChicago 
Consortium helps to build capacity for school reform by identifying what matters for 
student success and school improvement, creating critical indicators to chart progress, 
and conducting theory-driven evaluation to identify how programs and policies are 
working. The Network for College Success (NCS) partners with a cohort of Chicago's 
public high schools to dramatically increase high school graduation, college enrollment, 
and college success rates for all students. NCS helps schools respond to emerging 
research and the latest data to create actionable strategies to organize schools for 
improvement and prepare more students for the rigors of college. The To& Through 
Project is an initiative that utilizes NCS and the UChicago Consortium as partners 

to integrate research, data, and professional learning to move more students to and 
through high school and college. Together, these three entities have worked both 
independently and together to help educators and leaders at all levels of CPS use data 
as a tool to guide improvement in the outcomes that matter most for students. 
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It is important to understand the context in which the lessons we describe in this paper 
were generated. First, our role as partners to CPS has granted us an outside perspective 
and flexibility that are different from the context of those working inside districts 

or state agencies. This perspective gives us the advantage of seeing the public system 
from a slight remove, though the lessons we draw here may vary somewhat from the 
conclusions someone inside the system would make. Second, this paper focuses on the 
lessons we have learned specifically from doing this work in high schools, where the 
relationship between the three partner entities and CPS has been deepest. Though we 
believe that many of the lessons apply outside of high schools, we also acknowledge 
that the high school context is different than either K-8 or post-secondary education. 
Finally, the approach we describe in this paper has been generated in a distinctly place- 
based partnership, and all of the examples presented here reflect our work in Chicago. 


That said, as we continue to broaden our reach to schools, districts, and states across the 


country, we find that these lessons do apply in other places. 


The Chicago Public Schools 


(2017-18)* 


The Chicago Public Schools (CPS) is the third 


largest school district in the country and home to 


a diverse array of schools, students, and educators. 


The approach to using data to guide educational 
improvement that we describe in this paper was 

developed in the Chicago context. Here are some 
important facts about CPS that help explain that 


context: 


Schools: There are 165 high schools in CPS, 
including 92 district-run high schools, 65 
charter schools, seven contract schools, and one 
SAFE school. CPS has 479 elementary schools, 
including 421 district-run, 56 charter schools, 
and two contract schools. One of the unusual 


characteristics of CPS elementary schools is that 


the vast majority serve students from kindergarten 
through eighth grade, and there are very few 
dedicated middle schools. 


Students: In grades 9-12, CPS has 107,352 
students, 38.3 percent are Black, 47.25 percent are 
Latino, 8.6 percent are White, 4 percent are Asian, 
1 percent are multi-racial, 0.3 percent are Native 
American/Alaskan, 0.2 percent are Hawaiian/ 
Pacific Islander, and race/ethnicity is not available 
for 0.5 percent of high school students. CPS serves 
244,589 students in grades K-8. 


Teachers and principals: Across all CPS 
district-run schools, there are 19,324 teachers and 


505 principals. 


A Retrieved from: www.cps.edu/About_CPS/At-a-glance/Pages/Stats_and_facts.aspx. 
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It is our hope that the lessons shared in this paper about how preparing educators to 
do the hard work—focusing on the indicators that matter most; making meaning to 
build collective ownership of a challenge; strategizing to use the right data at the right 
time; and interrupting inequity—can help organizations, cities, sectors, and schools 
across the country understand how a practice-driven approach to using data to guide 
educator practice has been a critical component of Chicago’s remarkable success story. 
We hope the practice-driven data approach can further spark a deep dialogue to push 
the field toward a collective vision of creating lasting, impactful research-practice 
partnerships to improve school systems, resulting in students who are prepared for life 
after their K-12 education. 
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Using data to guide school improvement means that the conversation around the data 

is just as important as the data itself. For data to improve student outcomes, educators 
must be able to use data in conversations about their practice. Our work in Chicago 

has underscored the importance of building educator capacity to have hard data 
conversations that clarify what the problem is and what the solutions might be. This 
requires investing in the capability of one or a few people at a school to lead data-driven 
conversations, and it also requires strong school leadership to support a culture of data- 
driven improvement. The importance of this goal has clear implications for data work at 
the school level, but there are also less obvious implications for the design of the data 


ecosystem, as well as the research and analysis that underpins the work. 


School-Based Educators: 


Build capacity in teams around the technical ability to analyze 
data, and the adaptive ability to lead team conversations on data 


The practice-driven data approach relies on developing specific capacities, creating 
teams, and fostering important conditions within those teams and within the school. 
NCS, with professional learning and coaching, supports school teams to use data in 
four stages (see Figure 2); each stage depends on an interplay of analytic capacity, 
communications skills, and the ability to work with individuals and teams to act on the 
data. This work requires essentially two roles: a skilled data strategist and a team leader 
who collaboratively brings data to life in team meetings. Sometimes, these roles can be 
played by one person on a team who can execute the entire process, but more often, the 
process requires two different individuals building the right skills and working together. 
Whether the role resides in one person or two, NCS encourages schools to view these 
pieces of work—generating the data and driving the conversation—as two sides of the 


same coin. 


An effective team leader takes on the responsibility of supporting the development 
and executing the work of a high-functioning professional learning community. This 
requires the leader to build skills in facilitating data-driven discussions with a team. 
One of the foundational tenets of using data for improvement is setting the conditions 


for trust and collective responsibility so that the process is safe for individuals and 
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| Figure 2: Stages of Data Strategy in NCS Partner High Schools 
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Developed by Quinton Keith and NCS. 


teams to have hard conversations about data and improving their practice. In order to 
be effective, a team leader needs to be able to support the development of this trust on 
a team, usually by generating a clear mission to drive the team’s work; connecting to 
shared core values and intentions for serving students the best that they can; and using 
carefully structured protocols to introduce data into the conversation in a way that 
instills a sense of efficacy, rather than defensiveness, among team members. It takes time 
and skill for a team leader help a team build routines of analyzing student outcome data 
and using that data to interrogate current practices. The team leader will likely require 
some training and ongoing support to build and sustain these practices.” 


A school-based data strategist plays a critical role in the work of teams to use data to 
inform practice, although the nature of this role could vary across contexts. In some 
cases, a data strategist could be from an organization like NCS. A data strategist 
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could also be a stand-alone role within a school, with a job of supporting all areas 

of the school with strategic data support. More often than not, though, the role of 
data strategist is played by an educator already on a team who is able to strengthen 
his or her analytic skills to support the team in a new capacity. A school-based data 


strategist needs to learn which data systems or dashboards house the data the team 
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needs. The strategist also needs to know how to analyze the data so that it answers the 
questions that the team needs answered. Data strategists benefit from district-provided 
professional development on how to use school-facing data systems, and they can also 


benefit from forming professional learning communities where they can share tools, 
tips, and tricks for making the most of the data available to them and their schools. 
Data strategists must build fluency in various data analysis applications and skills (e.g., 
Microsoft Excel, programming languages, data visualization software, etc.) in order to 


execute statistical tasks efficiently and accurately. 


Whether working independently or in collaboration with a team leader, a data strategist 
goes through four phases of data analysis, as shown in Figure 2. First, the data strategist 
needs to build understanding of the purpose and need for data, as well as the intricacies 
of the data systems available. Second, the data strategist collects the data from the 
available data systems and, third, transforms the data into the representation that will 
best inform an educator's or team’s practice. Finally, the data strategist socializes the 
data, connecting with relevant stakeholders to make meaning of and act on the data. 
The interplay of each stage of this process pushes those analyzing the data to consider 
the way it will be used in the school (e.g., building awareness of the numbers with 
stakeholders, bringing data to teams for action, communicating the results to students 
and parents, etc.) as an integral step in data analysis. Figure 3 gives an example of the 
kinds of technical and adaptive tasks required at each stage of data strategy, in this case 
relating to how a data strategist would analyze student attendance data in different 
ways and with different people in a school. This multi-step approach to data strategy 
underscores that, just as data analysis itself takes time and care to complete, so too does 
planning a productive conversation around the data. 


| Figure 3: Stages of Data Strategy Applied to Attendance Data 


Developed by Quinton Keith and NCS. 
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Create summary graphs 
and bullet points of 
important observations 
for senior leadership 
team. 


Create a graph that 
combines attendance 
data and on-track data 
of Ninth-Graders for 
the Freshman Success 
Team. 


Create a roster of 
chronically absent 
students for Care Team. 


With principal, pick an 
appropriate professional 


reading on attendance to 
review and discuss with 
senior leadership team. 


With Freshman Success 
Team lead, pick 
appropriate protocol 

to structure data 
conversation. 


Support the Care Team in 
making a plan to follow 
up with chronically absent 
students. 


DuFour & Eaker (1998). 


| Table A: Common Teams in NCS Partner High Schools 


Team Name 


Team Members 


Team Responsibility 


Data Used 


Freshman 
Success 
Team 


Freshman teachers, 
counselors, and 
other educators; 
administrator 
supporting the work 


Monitoring students’ 
transition into high 
school and supporting 
their academic 
achievement in ninth 
grade 


GPA, course grades, 
attendance rates, and 
behavioral data for 
freshman; eighth-grade 
early warning indicator 
data 


Instructional 


Teacher leaders from 


Ensuring instructional 


Student assessment 


Leadership across the building; coherence and pushing data; student course 
Team administrator toward instructional performance data 
supporting the work improvement 
Postsecondary Counselors, teachers, Supporting students’ Data on students’ 
Leadership and other educators; college search and college applications; 
Team administrator application process, as student academic 
supporting the work well as their eventual performance data; 
transition into college; college enrollment data 
creating a school-wide from previous cohorts 
college-going culture 
Senior Leaders and Setting a vision for the High-level data from 
Leadership administrators from work and monitoring all other teams 
Team all areas of school progress toward 
functioning important outcomes 
Care Team Dedicated specialists Intervening with Attendance and 


for supporting 
socially vulnerable 
students—social 
workers, attendance 
interventionists, 
counselors, health 
professionals, etc. 
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students whose 
social, emotional, or 
physical needs are the 
deepest—those who 
are not easily reached 
by other educator 
teams in the building 


behavior data; data 
from students 
Individualized 
Educational Plans 


A common collaboration rhythm for school- 
based teams (like Freshman Success Teams and 
Postsecondary Leadership Teams) is a bi-weekly 
meeting that lasts approximately 45-60 minutes. 
During this time, teams analyze student data, 
set strategies for interventions and supports, and 
progress monitor or reflect on the effectiveness 
of interventions to date. These tasks happen in 
addition to the ongoing and necessary work of 
a professional learning community, like making 
meaning of professional readings and looking at 
student work together. 


In addition to developing the capacity of team 
leaders and data strategists to facilitate data- 
driven conversations in teams, the principal plays 
a crucial role in shaping the culture of a school 

to use data for improvement. High-functioning 
teams are rarely formed in isolation at a school or 
in contrast to the school’s culture. Rather, high- 
functioning teams flourish where the school-wide 
culture, systems, and structures are in place to 
drive the work. One of the hallmarks of effective 
data use in schools is the investment in a process 
that Janet A. Weiss describes as “making data 
safe.” “ Many educators have primarily experienced 
data only punitively, as a way to make harsh 
judgments about individual teachers or students. 


C Weiss (2012). 


Part of the role of the school leader is to not 

let this history stand between current practice 

and improved practice, and that journey nearly 
always requires the use of data to have hard 
conversations. The goal of approaching data safely, 
therefore, is not to make the data itself nice or 
palatable to educators, but rather to build the 
conditions necessary for team members to have 
hard conversations with each other about current 
practice and strategies for improvement. 


A great deal of trust-building needs to occur 
within a team and across a school in order to make 
the conditions safe for teachers to use data to 
collaborate and improve. Though some tools can 
help—norms, protocols, and listening techniques, 
to name a few—there is no standardized approach 
to developing trust. The work of trust-building 

is deeply contextual and personal. It is not built 
in an off-site professional learning session, but 
rather can only be developed by engaging in 
difficult data-driven conversations, and then 
reflecting on that process. It starts at the top of 

a school building. Fundamentally, principals 

and administrators must set the conditions for 
individuals and teams to trust each other in order 
to keep a lens of improvement and support, rather 
than shame and judgment. 


When teams have these elements in place—the technical ability to analyze data; the 


adaptive capacity to lead team conversations on data; and a trusting, supportive school- 


wide culture that promotes data as a tool for improvement—they begin collaborating 


at a high level. Data can be used to highlight variation in outcomes across educators 


and support them in creating the conditions necessary to push each other toward better 


and more equitable outcomes for all students. In high-functioning teams, educators are 


motivated to push for better outcomes for their students by their commitment to their 


students and their collective responsibility as peers and not from an external motivator, 


like a rating system. Teams that can do this—use data to hold their colleagues 


accountable to their shared responsibilities and student goals—are characterized by a 


remarkably high level of social trust, which takes time and intention to build. 
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Data System Designers: 


Provide cross-school analysis and give educators dedicated time 
and space to learn from each other about promising practices at 
different schools 


Data systems can play a crucial role by providing information in ways that can inform 
conversations across schools—both in how the systems are technically designed to 
display data and in how those data displays are designed to generate conversations 
among educators. Schools are the primary lever for change in a school system, 

but learning across schools can catalyze changes in practice within schools. Most 
importantly, effective data systems show comparisons of indicators and outcomes 

of similar students across schools in order to highlight differences in practice, not 
differences in population, which is how data is typically presented. This kind of data 
disaggregation allows educators to more critically examine their own strengths and 
weaknesses, as well as to seek out peers who are achieving impressive results with 
similar populations and who may have promising practices to share. For example, 
Consortium research on middle grade indicators of high school graduation identifies 
eighth-grade GPA and eighth-grade attendance as the most powerful predictors of 
ninth-grade performance.” For that reason, the To& Through Online Tool (a public- 
facing data tool with data on all Chicago public high schools) disaggregates freshman 
course performance by those indicators of prior performance, combined into summary 
measures that estimates a student's risk and opportunity for success or struggle in ninth 
grade.’ The tool also provides schools with data on how “comparison schools” (schools 
whose ninth-grade students had similar academic performance in the middle grades) are 


faring so that schools can put their results into context. 


| Figure 4: Sample School Freshmen who are On-Track with a GPA 2 3.0 
by Risk and Opportunity Group 


Fewer than 10 of high risk freshmen are on-track with a GPA of 3.0 or above 
Not shown: Fower than tO students in the group 


2 310% cps 
+ 7% at Comparison Schools 


22% of vulnerable freshmen are on-track with a GPA of 3.0 of above 
. > 27% at Comparison Schools 
218% CPS 
42% of opportunity freshmen are on-track with a GPA of 3.0 of above 
: + 42% at Comparison Schools 
+38% CPS 
41% of high opportunity freshmen are on-track with a GPA of 3.0 or above 


51% at Comparison Schools : : 
56% CPS; 


Visualization adapted from the To&Through Project website. 
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Once the outcomes are disaggregated by students’ prior achievement, educators can 
compare the outcomes of similar students across schools. One frustrating element of 
typical cross-school data discussions is that outcomes are compared across schools that 
serve very different populations—for example, a neighborhood high school in a deeply- 
impoverished area compared to a selective enrollment school where students have to 
demonstrate high achievement in order to be admitted, or schools serving very different 
populations across a state. Focusing on students at a particular band of academic 
achievement—for example, freshmen entering high school at high risk for course failure 
or seniors with qualifications to attend a highly-selective four-year college—allows 
educators to make more rational and fair comparisons across context. This, in turn, 
removes some quick explanations or defensive excuses about why schools’ outcomes are 
different and pushes educators to talk about differences in practice, not population. 


This practice of highlighting schools that are making 
progress can reframe the conversation from one that 
is focused on negative perceptions of students to one 
that lifts up the responsibilities of adults. 


In Chicago, a system with over 90 district-run high schools serving dramatically 
different populations, this approach to disaggregation of data also supports the search 
for best practices, allowing teams to discover areas of relative strength and weakness 

for a given population compared to other schools. As the work of supporting students’ 
transition to high school has evolved over time, one group of students that has emerged 
as important is “high opportunity students,” ninth-graders with strong eighth-grade 
course performance. For the school in Figure 4, only 41 percent of these students 
earned a 3.0 GPA in freshman year at the school, a result that is lower than the results 
for high opportunity students citywide or at similar schools. This information allows 
the school in Figure 4 to search for comparison schools that have better outcomes with 
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“high opportunity students” and engage in powerful cross-school practice sharing. 


However, to make use of disaggregated data and cross-school comparisons, the 
conditions for adult learning need to exist. Educators require dedicated time, space, 


trust, and safety (discussed within schools, above) to have constructive conversations 


auvdidd 


across schools to share practices. Supporting effective cross-school conversations on data 
requires careful attention to objectives, norms, and facilitation from a lens of authentic 
improvement. School leaders may never be able to create a learning space entirely 


separate from accountability in their schools, but by positioning themselves as learners 
and looking to educators as professionals and experts, they can find ways to hold space 
where data for improvement takes center stage. Consider, for example, the High School 
Institute, a networked learning space jointly hosted by the To& Through Project, NCS, 
and the Chicago Public Schools. An Institute session might begin with the research 
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about a given indicator’s relationship to a student outcome, grounding the conversation 
on why the particular indicator is relevant to student success. Then, student outcomes 
on one indicator can be displayed for all schools at one time, with school teams having 
space and time to engage in data-based inquiry. After teams discuss their data, educators 
can learn about practices from other schools that are showing improvement, giving 
those schools a chance to highlight a specific practice that has driven their data. 


Focusing on schools sharing their promising practices, even when those practices are 
not perfect, builds a spirit of collaborative problem-solving that fuels the spread of 
innovation. This practice of highlighting schools that are making progress can reframe 
the conversation from one that is focused on negative perceptions of students to one 
that lifts up the responsibilities of adults. When done well, this type of disaggregation 
and cross-school sharing can lead to an ethos that, regardless of what kind of schools 
educators work in—selective enrollment, neighborhood, South Side, North Side—they 


can learn valuable lessons from anyone in the city. 


Researchers and Analysts: 


Validate the problem of practice, identify strategies, and show 
variation across schools 


Schools are encouraged to use evidence-based strategies, and researchers can structure 
their studies to more effectively support the needs of educators. In order to build the 
kind of data systems that support this within- and across-school capacity to improve, 
the research standing behind the data will be most effective if it can provide three 
things: 


1. A clear understanding of the nature and scope of the problem of practice so 
educators can have conversations about the purpose and need for data in their school 


teams; 


2. Guidance on how to address the problem of practice so that educators are grounded 
on how to use the information contained in the data once educators have made 


meaning from it; and 
g 


3. Evidence of variation in student outcomes across schools so that educators identify 


and share effective strategies. 


The starting point for research and analysis should be to clarify what the problem is and 
what students and schools are affected by the problem. For example, when Consortium 
researchers began to examine dropout rates and who dropped out from high school, a 
key insight was that the majority of students who dropped out of high school were not 
in the lowest quartile of eighth-grade achievement.’ Defining the issue of high school 
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dropout as being not just about academic ability was an important insight for educators 
about how they might improve graduation rates. 


Knowing the nature and scope of the problem of practice is not sufficient; educators 
need to have research and analysis that enables them to act on the information 
contained in the data. After Consortium research demonstrated how higher-achieving 
students also dropped out of high school, the strategies for addressing this problem 

of practice pointed to a direction other than intervening with the lowest-achieving 
students or the students who had the deepest challenges. A subsequent Consortium 
report, What Matters for Staying On-Track and Graduating in the Chicago Public Schools, 
provided evidence on how poor attendance was a key driver of failure rates and not 
graduating from high school.” Armed with the knowledge that attendance was closely 
tied to graduation rates, schools developed strategies to address a problem of practice 
that was more manageable. 


Going deeper into the data to understand cross-school variation can help show schools 
how they have agency to move student outcomes. For example, in the same research 
report, Consortium researchers found that even when controlling for a number of 
considerations like the demographic and prior achievement data of CPS high schools, 
rates of ninth-grade course failure were substantially different across high schools. 
Furthermore, researchers identified several measures of school culture—especially 

those related to student-teacher relationships—that were associated with better-than- 
expected Freshman OnTrack rates."° This provided an important evidence-based theory 
of action to schools: if they worked on building better relationships between students 
and teachers, they might expect to see higher Freshman OnTrack rates at their school. 
By surfacing leverage points and demonstrating that there is sufficient variation in 
outcomes across schools, schools can work on problems and share ideas in networked 
improvement communities about promising practices. When educators can see clearly 
that students like the ones they serve do better at other schools—and when they can see 
that there are clear levers for improvement—they take ownership of student outcomes 


in a way that fuels their need for and use of data. 
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In a relatively short period of time, CPS, like a great many school districts across the 
country, moved from a system in which educators were thirsty for any data to one 
where they could drown in data if they were not careful." The proliferation of data meant 
educators needed to find ways to focus their discussions on the most important data. It 
is the role of researchers to work with educators and develop high-leverage indicators 
that can help them focus on what matters the most for future student outcomes. Data 
system designers can then integrate these indicators into the data that reaches schools. 
Finally, educators and school leaders can then incorporate this data into their ongoing 
work, using these indicators to track progress and examine patterns both within and 


across schools. 


School-Based Educators: 


Prioritize indicator data at the student-level and the school-level, 
while using other data points as context 


One of the most valuable resources in schools is time, and indicators can help educators 
prioritize how they spend their time. For schools inundated with information and data 
systems, research-backed indicators help educators make sense of the existing data as 
part of the regular process of education. Knowing which data points are crucial for 
students’ long-term success helps schools to make decisions about what to emphasize 
over the course of a school year. Specifically, schools can use indicators at two primary 
levels: first, at the student level, to identify who needs support and intervention; 

and second, at the school level, to target, guide, and assess progress of improvement 
strategies. When used effectively, indicators are integrated into daily practice and are 

a part of a highly-collaborative environment that can bring together teachers, school 
leaders, other school staff, families, and students to work together on common goals. 
Meaningful and malleable indicators set the conditions for more positive, solution- 
oriented conversations by keeping the focus on outcomes that educators believe matter 
to the students they care about—and outcomes educators believe they can change. 


This focus does not necessarily mean talking about a single indicator and no other data, 
nor does it mean analyzing the indicator in the same way over and over again. School 
teams that have built some interpersonal trust and data analysis skills might choose to 


review a wide range of variables at a team meeting. The key is to lead with the indicator 
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| Figure 5: Intervention Flowchart Addressing Freshman GPA Decline 
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Developed by Kareem Sayegh and the Freshman Success Team at Sarah E. Goode STEM Academy. 


and build other data into the conversation for context or explanation. For example, a 
Freshman Success Team will probably check a point-in-time Freshman OnTrack rate 
as a first pass at their student data at a meeting, but then they will likely use other data 
to help understand the nature of students’ challenges and make strategies to better 
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support them. Looking at Freshman OnTrack rates helps answer the question of how 
many students are off-track. Yet, Freshman Success Teams also need to ask a series of 
additional questions to guide their problem-solving. How many students are failing 
courses because they are not attending regularly versus attending regularly and still 


failing? How much is failure driven by poor grades on large exams or assignments versus 
failure to hand in homework or participate in class? How many students are failing a 
class for the first time in their lives and how many have a past history of course failure? 
To answer these questions and others, teams need to be able to review data in stages to 


guide their problem solving. 
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The flow chart in Figure 5 describes how the Freshman Success Team at Sarah E. 
Goode STEM Academy in Chicago thinks about developing interventions for freshmen 
who are experiencing GPA declines in the transition to high school. This chart gives 

a sense of the kind of triage process that schools can develop using a staged approach 
to data. First, the team distinguishes between students with extremely low attendance 
and those who are coming semi-regularly, referring the students with the deepest 
attendance challenges to a team of attendance specialists. Then, the team uses GPA as 
a marker to distinguish between students having different kinds of academic struggles, 
developing interventions both within and outside of classrooms to respond to students’ 
needs. Other schools might also use data to distinguish between students with and 
without discipline infractions, students who are or are not involved in particular school 
programs, and/or students in different curricular tracks or cohorts within a school. 


This kind of data analysis—introducing the data at different stages to assign students 

to different interventions—is crucial for problem-solving and helps to make a task like 
developing and monitoring interventions for a large number of freshman feel more 
manageable to educators. In addition, note that the assumption in the flow chart, as 

it should be in indicator use in general, is that the predictive power of indicators is 
practical rather than deterministic. Indicators are used to estimate a student's likely 
outcome, determine whether a student is in need of additional supports, and then work 
to change the student’s outcomes for the better. That is, the goal is to take action so that 
the predicted outcome does not correspond with the student’s actual outcome. 


Data System Designers: 


Prioritize how school-based educators can most readily use 
indicator data for improvement 


While schools require flexibility in day-to-day data monitoring to meet their immediate 
practice needs, the work of looking at student outcome data over time or generally 

in the “big picture” view is better served by simplicity rather than complexity. When 
data systems are not focused, the implicit message to educators is that everything 

(and therefore nothing) is important. The best way to accomplish this simplicity is to 
prioritize school-based educators above other audiences and research-based indicators 
above other data points. Here are four guidelines to focus and align data systems while 


avoiding common mistakes. 


Streamline the amount of data — For many districts and states, the default 
approach to data systems is: “If we have the data and someone might want to use 

it, we ought to include it.” When data is available on everything from students’ 
attendance to students’ clicks in a software program, educators can spend valuable 
time sorting through tables and graphs that have no connection to students’ outcomes. 
It is important to streamline and simplify data for educators by asking: What are the 
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most important questions we want school-based practitioners to be able to answer and 
what data do we absolutely need to provide to enable them to answer those questions? 
Placing these questions at the top of any data set or display can clarify what the focus 
should be for educators. 


To achieve simplicity, prioritize school-based educators 
above other audiences and research-based indicators 
above other data points. 


Show data in ways that make it easy to understand — It is critical that data 
be understood easily so that practitioners can quickly move to conversations about the 
data. Although complicated data displays can provide a lot of nuance and be visually 
interesting, users on the ground prefer something clean and simple. This allows them 
to go right to the information they need without getting lost in the esoteric differences 
between data displays. The default should always be to display the data once or twice 
and to choose displays with which educators are already familiar. In order to justify a 
new way of presenting data, there should be clear evidence that the benefits of the new 
format considerably outweigh the expected learning curve for school-based practitioners 
to understand how to interpret the new format. Basic stacked bar and line graphs, as 
shown in Figure 6, are often the best way to display indicator data. 


Consolidate data into one system — Even the most intuitive and focused data 
systems are not helpful if the data are held in separate, disconnected interfaces. The 
onus then falls on the front-line user to integrate data across systems in order to answer 
seemingly simple questions about indicators. Data systems often evolve in different 
departments within a school district, and as a result, users need to take the time to 
access and merge data across multiple data systems. A lack of data interoperability sends 
the message to school-level practitioners that their work to improve different indicators 
is unrelated. The objective of the data system should be to, as much as is feasible, reduce 
the time and energy it takes for a school-based practitioner to integrate data from 
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different sources. The work of building data infrastructure to connect data points and 
products is essential for a data system’s utility for front-line users. 


Design data systems for practice, not accountability reporting — Data 


collection that occurs as the enforcement mechanism for a policy goal creates fatigue 
and can lead to mistrust in a school. Typically, this kind of data is collected for the 
primary purpose of informing a particular department or program about whether 
people on the ground are complying with mandates to implement a new curriculum, 


pedagogical approach, or system of support. The data collection itself is the way that 
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a school demonstrates compliance with the new policy or idea. Data in these cases are 
generated in the field and funneled up to higher levels of administration, and there is 
often no mechanism for educators to review the data themselves for their own inquiry. 
This kind of data collection wastes time and can reinforce the viewpoint of many 
educators that collecting and reporting data serves no purpose in their own work but 
is rather something they must do to accommodate district or civic leaders far removed 


from the work of schools. 


Figure 6: College Enrollment Rates for Moderately Qualified Graduates over Time 
at Sample School 


2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 


@ Moderately Qualified Graduates enrolled in a 2-year college 


@ Moderately Qualified Graduates enrolled in a 4-year college 


Visualization adapted from the To&Through Project website. 


Looking at one graph from the To&Through Online Tool in Figure 6 shows how 
specific data choices can prevent these mistakes. Notice that the graph begins with 

a question that frames the purpose of the data for the user, and that it is a relatively 
simple stacked line graph. The graph allows for disaggregation by a particular group 
of students, here focusing on moderately qualified graduates (or those with access to 
somewhat and non-selective four-year colleges). The display also integrates high school 
data and college data from different sources and is designed to push schools to raise 
questions about how shifts in practice during specific years may have led to changes in 


student outcomes. 
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Schools are awash with data. Without research evidence on which indicators to pay 
attention to, setting priorities and effectively focusing school resources and efforts 

can be extremely difficult. By organizing readily available information on student 
performance into indicators, educators can identify which students need what types 
of support and develop and test school strategies to improve students’ educational 
attainment. At its core, an indicator provides a prediction of a future outcome such as 


high school graduation. 


However, in developing indicators for use in schools, it is essential to pay as much T 
attention to how they will be incorporated in daily practice and used to make strategic 
decisions as their predictiveness. The role of researchers is not just to conduct studies 

to identify indicators that are highly predictive of critical educational outcomes, but 

to consider the role they will play in the school context and how they will be used 

(see box titled Characteristics of Effective Indicators). The power of the Freshman 

OnTrack indicator comes from much more than it being more predictive of high 

school graduation than eighth-grade test scores, SES, race/ethnicity, gender, and school 

mobility combined. Rather the power of the Freshman OnTrack indicator comes from 

utilizing readily available and understandable data to ground and focus problem-solving 


discussions within schools about student performance. 


Characteristics of Effective 
Indicators 


A recent Consortium report outlined 

five characteristics of effective indicators: 

1) predictiveness, 2) usability and clarity, 3) real 
time/right time availability, 4) direct causal 


linkage to outcomes, and 5) malleability. ° 


It is critically important that researchers provide 
evidence of what indicators are predictive and not 
predictive of the outcomes that schools seek to 
change. When deciding which indicators to use 

to guide practice, it is important to know which 
indicators are most predictive of students’ future 
success—and which indicators may be related to 
down-the-line outcomes, but are demonstrably less 
predictive than others. As important as knowing 
what indicators are useful in knowing how to 
target strategies and interventions, is knowing 
what indicators are less effective targets for 
improvement efforts and are not good measures of 
progress. However, predictiveness is still only part 
of the equation when deciding which indicators to 
use. Even with rigorous evidence on the predictive 
nature of an indicator, systems will still struggle to 
use indicators to focus if the indicators themselves 
arent usable and clear. Put simply: the analysis 

can be complex, but the best indicators are able 

to drive practice because they are understandable, 
simple, and can be applied in schools. When 
educators use indicators, the priority is to take 

in information quickly and to make indicators 
available in real time or at the right time so that 
action can be taken in a timely fashion. If the 
complexity in understanding and interpreting an 
indicator becomes a barrier to action, it will not 
be effective in practice. This endpoint should be 

a major consideration in indicator development. 
If the indicator is only available once a year, it 
may be useful in planning, but is not going to be 


D Allensworth, Nagoaka, and Johnson (2018). 
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effective for educators who need to take action on 
a regular basis in schools. This attention to clarity, 
usability, and timeliness in developing indicators 
streamlines data and keeps all elements of a data 
system focused on the outcomes that matter most 
for student success. 


Ideally, indicators should be used to develop 
school strategies that have a direct causal linkage to 
outcomes, meaning that the indicators are not only 
useful in identifying students for support, but also 
useful as a focus for changes in school practice. 
Establishing causality is a complicated task. Yet, 
when educators expend their resources to improve 
a specific indicator, it is essential that those 
practices change the ultimate outcome of interest. 
For example, Freshman OnTrack is often thought 
of as an early warning indicator to identify 

which students require intervention to improve 
their likelihood of graduating from high school. 
However, it is even more useful as a focus for 
school strategies because changes in school practice 
that improve Freshman OnTrack rates also lead to 
increases in high school graduation rates. 


Finally, it is critical that the indicators that guide 
the data work are malleable and actionable (i.e., 
that it is possible to move the indicator and 

that there are known strategies for moving the 
indicator). For example, even if eighth-grade test 
scores are useful to identify students who are likely 
to struggle academically in ninth grade, high 
school educators cannot change past performance. 
In contrast, freshman year course performance is 
something that educators can develop strategies 

to address, and research has demonstrated how 
attendance, for one, is a strong predictor of course 


performance and high school graduation. 
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When using data for improvement, accessible research findings give school and district 
leaders the opportunity to infuse their data work with research evidence. This, in turn, 
builds educators’ sense of ownership over the problems of practice raised by the 


research and commitment to changing adult practice to improve student outcomes. 


School-Based Educators: 


Facilitate educators’ understanding of the research behind 
the indicators 


Important ideas in educational measurement often move from being the answer to a 
research question to a research-backed indicator to a system-wide call to action to a 
school-level accountability metric. This process happens often without the people on 
the ground in schools getting a chance to make sense of the ideas behind the numbers. 
To move so quickly from the ideas to the data misses opportunities to work with 
teachers and other educators to generate their understanding of a problem, how the 
problem plays out in their context, and how they weigh the benefits and drawbacks 
of different strategies. 


An example of this comes from the UChicago Consortium’s work on college access. 

In From high school to the future: Potholes on the road to college, researchers found that 
students who completed three or more college applications were more likely to gain 
acceptance to a four-year college.” As a result, the proportion of seniors who completed 
three or more college applications became a component of school evaluation. As 

the idea moved from an illustrative research finding into an evaluative metric, some 
important context was lost in translation for many educators working on college access 
in schools. The biggest take-away of this line of research was that scores of motivated, 
talented CPS seniors fail to enroll in college every year simply because they do not 
know how to navigate the college application process—and that schools that set up the 
systems and structures necessary to support students in that process got good results. In 
this context, a school leader might consider monitoring the proportion of students who 
complete three or more college applications as an indicator of whether or not the school 
is meeting the broader imperative of supporting all students to achieve their post- 
secondary goals. Divorced from the context, the indicator felt arbitrary and misaligned 


to many educators. 
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When indicators reach educators without the research as context, educators have a 


tendency to approach indicators from a perspective of complying with mandate, rather 
than useful data to solve an important problem. By contrast, when educators are able to 
make their own meaning of the research findings, they are able to approach the work of 
moving student outcomes from a place of conviction, rather than a place of compliance 
with a policy. Working from a place of belief is far more likely to lead to long-lasting 
improvement efforts. Using the primary research in schools can be a powerful tool to 
build commitment, community, and collective responsibility for change in a school. 


Consider, again, the process by which high schools in Chicago began to organize 

their strands of work on the goal of sending more students to college. Driven in part 
by Consortium research on students’ post-secondary outcomes, the district began 
measuring and reporting college enrollment rates of CPS graduates in 2005. It was not 
immediately clear to the educators supporting students with the college process where 
the major points of leverage in their practice were—or if it was truly fair to hold high 
schools accountable for the college choices of their students. Over time, due to outreach 
efforts by the UChicago Consortium and facilitated learning opportunities facilitated 
by NCS, many educators were able to make their own meaning of important research 
reports like Potholes on the Road to College." This allowed them to see the current 
conditions that either produced roadblocks for students seeking to attend colleges (such 
as relying on students to seek counseling support when they needed it) or provided 
important supports (like developing a school-wide approach to college counseling). 

It also helped educators connect with the broader moral imperative behind the push 
to increase college enrollment numbers. Why were educators working so hard to prepare 
students for college academically if all of their efforts fell apart when students struggled to 
navigate the college application process? Using the research to understand strategies for 
improvement and build commitment to the work, in turn, helped educators develop a 
sense of ownership over their own process of building a college-going culture and the 
systems and structures to support all students to access to higher education. Working 
from this stance cements educators’ commitment to sustaining the hard work of 


incremental improvement over time. 
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Data for Improvement vs. 
Data for Accountability 


It is important to consider whether the ecosystem 
educators work within is designed to use data for 
improvement or for accountability. This distinction 
is not just about the data itself, but also about 
the purpose and conditions by which data is 
created and socialized. Data for accountability 
uses external incentives to signal which outcomes 
schools and teachers should prioritize.” Data for 
improvement focuses on creating the internal 
conditions that enable educators to use data to 
improve student outcomes in spaces that are safe 


from external pressure. 


It is a mistake to focus solely on either function 
for data. Most commonly, school systems 
prioritize data for accountability, providing little 
in the way of infrastructure, data, or support for 
improvement cycles, and assuming that schools 
have the capacity to generate and use aligned 
data for improvement. At the same time, it is 

not rational to think that data for improvement 
can be kept completely separate from data for 
accountability. Accountability systems are a part 
of every system of public education, and the lines 
between data for accountability and data for 
improvement will always be blurry: an indicator’s 
presence in an accountability system means that 
data will never become entirely safe from the 
judgments that come with accountability in public 
education. The pressure that educators feel to 
improve their schools’ accountability metrics is 
palpable—we still close schools, fire principals, 
and choose schools for our children based on 
this data. In addition, much of the data available 
to school-based educators are data generated 

in alignment with the accountability system, 
leaving savvy leaders in the position of retrofitting 


accountability data for improvement purposes. 


E Loeb & Figlio (2011); Ed Trust (2016). 
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However, the mere presence of data for 
accountability does not preclude an education 
system from creating authentic spaces where 
schools can use the data for improvement. Ideally, 
there is alignment between a system's data for 
accountability and data for improvement: the 
former should signal to schools what indicators 
they should focus on, while the latter makes 

up the bulk of the day-to-day work of using 

that indicator. For example, a school district 
could incorporate college enrollment into its 
accountability system, signaling to high schools 
that they should organize around improving 

this metric. The data system could then be used 
to make a compelling case that the day-to-day 
work of supporting students through the college 
application process is educators’ best leverage point 
to impact college enrollment percentages. The 
data system could also be used to support schools 
by providing systems to track college applications 
and FAFSA completion, coaching schools in how 
to structure bi-weekly meetings to review the 
data, and working with schools to share promising 
practices and integrate contextual challenges into 


their data and work. 


Efforts to use data for improvement begin 

to create a culture where educators take data 

for accountability as a given, but where data 

for improvement becomes a core part of the 
culture. In other words, a way of being and 
doing. However, committing to using data for 
improvement requires committing to openness, 
reflectiveness, and some amount of risk-taking on 
the part of leaders who must be willing to learn 
alongside their systems and schools. 


Data System Designers: 


Ensure data systems explicitly draw connections between school- 
level indicators and the research behind them 


Of course, the process of encouraging educators to make meaning of research across 

a school system as large as CPS is quite labor-intensive. It is not feasible to expect 
researchers or even school support organizations to reach all relevant educators in the 
system in person. There are certainly lighter-touch ways to emphasize the importance of 
the research in data work, starting with the data system itself. One of the core functions 
of a data system is to explicitly make the case to educators that there is a connection 
between their day-to-day work, the indicators of futures outcomes, and the outcomes 
themselves. In order to do so, a data system should directly link its indicators to the 
underlying research that explains why a particular indicator is associated with desired 
outcomes for students. When educators understand how the indicators link to student 
outcomes, and then are convinced that they can influence those indicators, they are 
more likely to believe that working on the indicators is about improving outcomes for 


their students rather than accountability or compliance. 


The To&Through Project sets out to do this in several ways: Issue Briefs highlighting 
what drives high school and college attainment (e.g., Freshman OnTrack, grades, 
etc.) and what strategies schools are using to move the needle on these measures; 
Mythbusters designed to dispel some common misconceptions on what drives high 
school and college attainment; Data Insights, or infographics that illustrate why post- 
secondary attainment matters and what can help foster it; and a Communications 
Toolkit that contains a collection of resources created to support school leaders in 
communicating research and data to stakeholders. 


In addition, the To&Through Online Tool provides an explanation of each indicator at 
the top of data displays. 
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| Figure 7: Landing Page for the To&Through Project Website 


96% of Westinghouse’s 2016-2017 first-time freshmen were on-track to graduate from high school in 4 
years. 
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Students who are on-track at the end of ninth grade are nearly three times more likely to 
graduate from high school than students who are off-track. A student is on-track if he or 
she fails no more than one semester of a core course and earns at least 5 credits by the end 
of freshman year. Additionally, only students who earn a 3.0 GPA or above in high school 
have at least a 50 percent chance of graduating from a four-year college within six years. 


This Tool makes explicit links between the way the data is shown and the relevant 
research. This kind of connection is vital to building and socializing a data ecosystem. 
In addition to adding brief reminders on the screen, the tool can help spur interest 
among educators in going back to the primary research texts to start doing the kind 
of meaning making —whether on their own or with their peers—that helps to build 
commitment to improvement. Of course, the quality and accessibility of the texts and 
tools that educators will discover if they follow the trail back to the research matters a 
great deal in this process. 


In certain situations, it can be helpful to replicate descriptive research findings at the 
school level. This process allows school-based practitioners to directly link an indicator 
to an outcome in their particular context. For example, it is one thing to know that 
freshman-year GPA is predictive of graduating GPA and college access, but the 
importance of freshman-year GPA may not be evident until a school team reviews the 
relevant data on their own students. The To&Through Online Tool provides graduating 
GPA over time at the school level for three different groups of students: on-track 
freshmen with > 3.0, on-track freshmen with < 3.0, and off-track freshmen. The vastly 
different outcomes for these three different groups of students make a clear case at the 


school level for the utility of freshman GPA as an early warning indicator. 
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| Figure 8: Changes over Time in High School Graduation for Different Groups of 
Freshmen at Sample School 


High school graduation for on-track freshmen with a GPA 2 3.0 


2011 2012 2013 2014 2015 2016 2017 
2007-2008 38% of 2008-09 49% of 2009-10 37% of 2010-11 41% of 201-12 33% of 2012-13 35% of 2013-14 
freshmen freshmen freshmen freshmen freshmen freshmen freshmen 


@ Graduated with a GPA below 3.0, on-track with a GPA of 3.0 or above as freshmen 
@ Graduated with a GPA of 3.0 or above, on-track with a GPA of 3.0 or above as freshmen 


Visualization adapted from the To&Through Project website. 


High school graduation for on-track freshmen with a GPA < 3.0 


2011 2012 2013 2014 2015 2016 2017 
2007-2008 47% of 2008-09 43% of 2009-10 51% of 2010-11 51% of 201-12 51% of 2012-13 53% of 2013-14 
freshmen freshmen freshmen freshmen freshmen freshmen freshmen 
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@ Graduated with a GPA below 3.0, on-track with a GPA below 3.0 as freshmen 
@ Graduated with a GPA of 3.0 or above, on-track with a GPA below 3.0 as freshmen 


Visualization adapted from the To&Through Project website. 
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High school graduation for off-track freshmen 


2011 2012 2013 2014 2015 2016 2017 
2007-2008 16% of 2008-09 8% of 2009-10 12% of 2010-11 8% of 201-12 16% of 2012-13 12% of 2013-14 
freshmen freshmen freshmen freshmen freshmen freshmen freshmen 


@ Graduated with no GPA reported, off-track as freshmen 
@ Graduated with a GPA below 3.0, off-track as freshmen 


@ Graduated with a GPA of 3.0 or above, off-track as freshmen 


Visualization adapted from the To&Through Project website. 


Researchers and Analysts: 


Build a two-way dialogue with practitioners to strengthen 
communication and ownership of research findings 


Building educator or school ownership of student outcomes is the result of an 
intentional process of conducting research and communicating findings that are 
relevant and actionable. From the beginning, the UChicago Consortium has 
distinguished its work from traditional academic research by conducting its studies 
with the audience of educators and other education stakeholders in mind, while still 
maintaining quality and rigor. From the onset of a study, the approach is motivated by 
research questions informed by pressing problems of practice and policy that emerge 
from schools and the district. While the methods underlying the research are guided by 
best practices in academic research, the selection of methods is driven by answering the 
research questions in ways that can guide district and school leaders. 


The UChicago Consortium has created structures within its organization that are 
intentionally designed to ensure that researchers receive adequate feedback from the 
field about what challenges educators are struggling with and what research products 
will best support those working to overcome those challenges. The UChicago 
Consortium’s research agenda is developed through a process of working with key 


education stakeholders every five years to ensure its planned studies reflect needs 
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emerging from the field. Individual studies are developed in consultation with district 
administrators and researchers provide regular updates as findings emerge so that 
district leaders have the time to develop thoughtful and intentional strategies and 
policies that draw on findings. The UChicago Consortium has a steering committee 
that provides feedback on studies in progress. The UChicago Consortium also has a 
formal “no surprises” policy of working with district leaders to share research findings in 
advance of public release of research reports. Just as important as the formal structures, 
though, is the power of long-standing relationships between researchers and education 
leaders at many levels of the school district. Both the commitment of CPS leaders to be 
guided by the best possible evidence and the responsiveness of Consortium researchers 
to collaborate and share findings with these leaders when and where they are needed 
builds an ongoing line of communication, as well as a sense of mutual accountability 


between both partners to use the partnership to the best of its promise. 


These researchers also help educators feel a sense of ownership by delivering research 
evidence in ways that are understandable by a broader audience and written with 
educators in mind. This approach is built on the belief that research findings will 
only be used by schools and districts if they are speaking to the priorities and needs 
of practitioners and are communicated in ways that make their implications clear. 
The UChicago Consortium engages with practitioners and policymakers throughout 
its studies to ensure that findings are understandable and actionable prior to their 
public release. 


Importantly, the approach does not end with the publication of a report or happen 
only within the timeframe of a given study. In order to build educators’ ownership of 
research findings, researchers need to dedicate time to listen to how educators frame 
their most pressing problems of practice. Through this dialogue, researchers build 
their own understanding of the right questions to ask, the best ways to share findings, 
and the context that educators need to start acting on findings. Researchers learn 

from practitioners about how to interpret findings in the distinct context of different 
schools or how policies play out in the school level. Practitioners frequently have a 
deeper understanding of the nuances of data and potential pitfalls, whether it is the 
meaning of specific variables, practices in data entry and timing that can influence their 
interpretation, and differences between the summative data researchers usually receive 
and what the interim data schools use may look like. Whether by interacting with 
Consortium researchers directly at professional learning opportunities, reviewing short 
research summaries on their own time, or using the primary research texts to guide the 
professional learning of a school-based team, educators in Chicago have been able to 
make meaning out of important research studies that guide the field. 


36 PRACTICE-DRIVEN DATA 


Decision-making in schools varies from big-picture strategy to fine-tuning interventions 
with individual students. Each of these decisions—and everything in between—benefit 
from the right data at the right time. The data system should provide schools with 
different data for different levels of decisions, with researchers evaluating popular 
strategies across contexts to determine the potential for scale. 


School-Based Educators: 
Get data at the right level to the right people at the right time 


One of the most important elements in making data work effectively in a school is the 
ability to get the right data to the right people at the right level of detail at the right 
time. Often schools have a tendency to focus either very broadly on the school average 
of an indicator (i.e., our Freshman OnTrack rate is 78 percent) or very narrowly on 
the short list of students with the deepest challenges (i.e., the 10 students failing two 
or more courses). Focusing too much on either end of this spectrum of detail misses 
the rich nuance and opportunity for improvement between the two poles. There are 
certainly different constituencies within the school who need the broadest possible look 
at the data and other constituencies who need the list of “high-flyers.” For example, a 
senior leadership team would likely want to see the school’s Freshman OnTrack rate, 
along with other high-level metrics every five weeks, but attendance interventionists 
need lists of students with chronic levels of absence and monitor it frequently. 


However, much of the work of teams like Freshman Success teams or Post-secondary 
Leadership teams is to find the students at the tipping point of success and failure. For 
example, students who are failing only one class or coming to school regularly, but often 
missing first period. Finding these groups of students who need similar and relatively 
light-touch supports from teachers and counselors to overcome fairly routine challenges 
in high school can have a two-fold benefit. First, for an individual student, it can 
prevent small challenges from growing larger. Second, it is an efficient way of working 
to improve a school’s overall performance on any given indicator. This process of 
grouping and monitoring requires a team to break apart the school’s average Freshman 
OnTrack rate and examine it from different lenses. It also pushes teams to effectively 
group different kinds of students for in-class and out-of-class interventions. 
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Wes Vasy 


In addition to thinking about data at different levels of granularity, it is also important 


to consider which data is most pressing to review at which points of the school year. 
Figure 9 shows an example of how some senior leadership teams in NCS partner high 
schools have chosen to structure the data focus of their monthly meetings, emphasizing 
different datasets at appropriate times of year. 


| Figure 9: Yearly Arc of High-Priority Data for Senior Leadership Teams in 
NCS Partner High Schools 


MONTH DATA FOCI RATIONALE 
September Who’s in Our Building Schools need to start the year with an understanding of their 
(demographics and prior population. 


achievement of students) 


October Whole School Attendance & Schools will want to catch attendance and discipline issues 
Discipline (with a focus on early before they become bigger climate/culture issues. 
freshmen) 

Early Freshman Course Performance End of first quarter is the earliest possible time to estimate 

November and Whole School End of on-track rates; schools will want to identify off-track students 

(End of Quarter GPAs early and provide support; schools will also want to start 

Ist Quarter) pushing for B-averages. 

December School- and/or teacher- By December, schools should have administered their own 
generated common assessment assessments of academic achievement. 
data 

January College Application Data Many application deadlines are in January/February; it’s 


important for schools to check on college match. 


February Freshman OnTrack and End It’s important for schools to check the efficacy of interventions 
(End of of Semester GPAs to keep freshmen on-track and to keep the rest of the school 
Ist Semester) high-achieving. 

March School- and/or teacher- By March, schools should have administered a second 
generated common assessment assessment of academic achievement; it’s also a good time 
data; estimates of performance to check in on where the school is landing on performance 
on accountability systems policies. 

April College Enrollment; Freshman Prior to the end of the year, schools will want to check in 
OnTrack and GPA on acceptance rates, scholarships, and estimates of intended 


enrollment. Also important to check on 3rd quarter freshman 
success rates to check on the extent of the “third quarter 
slump” commonly seen at many schools. 


May Who’s In Our Building NOW? Checking in on where the school will close out the year on 
various metrics and goals provides a great opportunity for 
reflecting on the effectiveness of interventions/strategies and 
will be a springboard into planning for the next year. 


Developed by Quinton Keith, Rachel Steele, Tim Valenti, and NCS. 
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Data System Designers: 


Produce different data for different times and purposes 


School practitioners need data at the right time and in the right form to effectively 
use it to support strategy and implementation. End-of-year data can guide longer 
and forward-looking cycles to set priorities and identify strategies (e.g., school 
improvement plans) while real-time data can drive shorter cycles of reflection and 
strategy adjustment in reaction to new information about student performance. 


End-of-year data, trends, and longer cycles of reflection and 
improvement — Since school systems generally have the time to make end-of- 
year data more precise, the objective should be to enable users to conduct more 
sophisticated analysis on this data. In this way, the data can be used for high-level 
strategy decisions that may lead to significant shifts in practice. It can take several 
years to see changes in student outcomes (i.e., high school or college graduation), 
and since these outcomes vary at different types of schools, school systems should 
try to provide end-of-year data as part of longer historical trends. The more precise 
nature of end-of-year data means that it is also the form of data that is public facing, 
allowing a variety of stakeholders both within and outside of a school to participate 
in their own long-term strategy and implementation cycles. Given the sometimes- 
public role of this data and the year-over-year comparison, it is vital to maintain 
consistent data rules across time so that schools and school partners can reliably 


compare outcomes from one year to another. 


Real-time data and short cycles of reflection and improvement — On 
the other end of the spectrum is real-time data. By its very nature, real-time data 
places a greater emphasis on timeliness rather than data-quality. For example, data 
may not be as accurate because a teacher may not have updated grades recently; a 
student may have just transferred in or out; or an administrator is late for processing 
a long absence as “excused.” Though systems should try to limit these errors as much 
as possible, the often inexact nature of real-time data means that systems should 

also limit possible analysis to what they term the most important views of data. 

The purpose of this data is to support adjustments in implementation (e.g. tactics, 
interventions, moves), rather than starting an entirely new approach. 


In addition to a simple table or graph, it is incumbent on the data system to enable 
schools to easily access the names behind the real-time data in a roster format. 
Rosters are the link between a data system and student-level interventions or Multi- 
Tiered Systems of Support, and the organization of a roster should privilege the 
indicators a system wants its schools to focus on and use language consistent with 
quarterly and end-of-year data. In addition, the format of a roster should allow a 
team of adults to quickly filter and sort through different sub-groups to move further 


toward hypothesizing about potential root causes. 
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Figure 10 shows an example of a real-time roster that counselors and Post-secondary 
Leadership Teams use when working with NCS. The roster pulls data from Naviance, 
the software that CPS counselors use to monitor and support students’ college 
application activity. The roster organizes the information to show counselors what 
kinds of college applications students are submitting and whether or not students have 
hit the school’s targets, such as submitting at least three applications and at least one 
application to a “match” college.” Counselors—or other members of a Post-secondary 
Leadership Team—can use this roster to regularly check their student caseload and 

see which students to target in any given week on whether they are making effective 
progress toward their post-secondary planning goals. Rosters like these—especially ones 
that can be frequently updated to reflect meaningful changes in status—are critical tools 
for educators taking a case management approach to support a school-wide goal, such 


as meeting appropriately ambitious college application benchmarks.” 


Quarterly or mid-quarter data — Quarterly data falls in between end-of-year 

and real-time data. One distinction between real-time data and quarterly data is that, 
generally speaking, both schools and systems can take a little extra time and energy with 
quarterly data to ensure quality and may also find time for deeper analysis and broader 
socialization. Quarterly metrics also give educators a chance to step back and place their 
quarterly results in the context of year-over-year trends, while also doing some basic 
analysis to look at the results from different points of view. At the same time, is also 
helpful to focus on one or two groups of students to develop an understanding of what 
quarterly data looks like at the student level as well as to transition into planning for 


tiered intervention. 
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| Figure 10: College Application Tracker 


Application 
Application Counts by Match Range 
Thresholds 
Student’s Three One Match 
Student College Bete) Safety Match Reach Too coe, ERS Application Application 
Name Access Safe Selective Milestone Milestone 
Level Met? Met? 
Student 1 Somewhat Selective 
Student 2 Somewhat Selective 
Student 3 Selective 
Student 4 Selective 
Student 5 Somewhat Selective 
Student 6 Somewhat Selective 
Student 7 Selective 
Student 8 Two-Year 
Student 9 Somewhat Selective 
Student 10 Selective 
Student 11 Non-Selective 
Student 12 Two-Year 
o 
Student 13 Somewhat Selective m 
Cp) 
Student 14 Selective oe 
@) 
Student 15 Somewhat Selective 2 
aN 
Student 16 Somewhat Selective 
wn 
Student 17 Selective a 
a 
Student 18 Somewhat Selective f 
m 
Student 19 Non-Selective (9) 
Student 20 Non-Selective ~ 
Student 21 Two-Year 


Student 22 Somewhat Selective 


Student 23 Selective 
Student 24 Selective 
Student 25 Selective 


Developed by Nate Flint and NCS. 


43 


Researchers and Analysts: 


Develop multiple indicators that can be used for different purposes 


As schools engage in cycles of reflection and improvement, having multiple evidence- 
based indicators of the same outcome that can be used at different points in the cycle 
are essential. For example, the Freshman OnTrack indicator evaluates where a student 
is at the end of freshman year, and provides a useful metric for assessing the progress 

of a school on a yearly basis and develop priorities and goals for the next school year. 
The Freshman OnTrack indicator does not help educators understand where individual 
students stand at a given moment during the school year. In addition to the Freshman 
OnTrack indicator, Consortium research demonstrated that attendance and grades, 
particularly course failures, were also highly predictive of Freshman OnTrack status and 
high school graduation.” Grades and attendance are collected on an on-going basis in 
schools, and can be used by educators to identify students or groups of students in need 
of additional support and intervention. Thus researchers and analysts should be mindful 
of developing indicators that can be used for different purposes at different stages in the 


cycle of reflection and improvement. 


Another dimension of indicator development that researchers and analysts should 
consider is ensuring that indicators are reflective of multiple priorities and different 
populations, particularly achievement level. For example, while the Freshman OnTrack 
metric is highly predictive of high school graduation, it is less helpful in supporting 
schools in supporting their students in enrolling and succeeding in college, particularly 
for students who entered high school having strong eighth-grade attendance and 
grades. The UChicago Consortium developed the “risk and opportunity” categories 
and used both Freshman OnTrack and a 3.0 GPA as outcomes to reflect that students 
who had lower eighth-grade attendance and grades would likely need support to ensure 
that they were on-track, while students with stronger qualifications may need support 
to maintain their eighth-grade level of high performance and put them on a path to 
college readiness.” When researchers develop differentiated indicators for different 


students, educators are better positioned to help their students meet their potential. 
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Decades of public discussion on the need to reduce achievement gaps has done little 
to produce more equitable outcomes for American students. In Chicago, we use data 
not only to highlight differences in student achievement, but also to push educators 
to examine the beliefs, practices, and institutional conditions that create inequitable 
outcomes for our youth across the district. Everything from the intentionality of the 
conversations at the school to the organization of the data ecosystem to the design of 
the research itself has implications for equity. The approach we describe here creates 
the moral imperative for educators to make the changes in their practice necessary to 
change marginalized students’ experiences in public education. 


School-Based Educators: 


Critically examine the implications when different groups of 
students have different outcomes 


In her seminal address to members of the American Education Research Association 
(AERA), AERA President Gloria Ladson-Billings made a case to academics, 

advocates, and policymakers that we should move away from the familiar language 

of an achievement gap in American education and instead embrace the language 

and framing of an educational debt, owed to those communities of Americans from 
whom educational resources have been historically withheld. The speech provides a 
powerful reframing of a persistent problem and also shines a light on how the language 
embedded in the discussion on achieving equitable educational outcomes in America 


has been a roadblock for improvement in and of itself. 


In order to responsibly use data for improvement in schools like those in Chicago, it is 
important for school-based teams to push themselves to examine students’ outcomes 
by race, gender, income, English learner (EL) status, diverse learner status, and any 
other area of vulnerability students experience in the school system. However, the 
language, context, and framing of this data are fraught with potential pitfalls of the 
kind that Ladson-Billings warns about. Decades of school outcomes disaggregated. by 
these subgroups has produced important datasets and supported conversations about 
inequitable outcomes for students, but not all of these conversations have had the 


effect of spurring educators to seek new ways to support the learning of historically 
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marginalized groups of students. Indeed, the practice of continually, publicly 
documenting outcomes like high school graduation with the rates of White students 
compared to the rates of students of color can be demoralizing to communities of 
color. Presenting the data in this way—which emphasizes the differences in student 
outcomes without bringing to light the structures in educational systems that produce 
inequitable results—can sometimes lead educators to rely on their implicit biases for 
explanations. It can also subtly reinforce a problematic narrative that whiteness itself 
is a goal that communities of color should hope to achieve. Given these concerns, 
explicitly addressing disparate outcomes for marginalized groups at any given school is 
as important as it is perilous. 


Productive data-driven conversations about equity 
in educational outcomes focus educators on an 
examination of their own practice to understand why 
students experience such different outcomes in the 
present—in these classes and at this school. 


Productive data-driven conversations about equity in educational outcomes focus 
educators on an examination of their own practice to understand why students 
experience such different outcomes in the present—in these classes and at this school. 


It can be thorny to disentangle the extent to which student outcomes are driven by the 
kinds of vulnerabilities students bring with them to school (as a result of systematic 

and long-term inequities) versus the current practices of adults in their high schools. 
This is especially true in high schools, where students bring with them a 14-year legacy 
of school and life experiences. Nevertheless, this process is necessary for data work in 
schools. For example, in Chicago, Freshman Success Teams use different categories of 
students to case manage their freshman class from the beginning of the year. These “risk 
and opportunity” categories—which are based on eighth-grade GPA and attendance— 
have been validated as predictive of ninth-grade performance across schools and across 
subgroups of students, so they provide an important lens for planning and monitoring 
student outcomes in freshman year.” Freshman Success Teams also typically review data 


by subgroups like race, and when they do, they may notice that, for example, Black 
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young men may be less likely than their peers to earn a 3.0 GPA. The team may also 
notice that Black young men are less likely than their peers to have entered high school 
with a prior track record of high grades and strong attendance. If the conversation ends 


here, then the team has let itself off the hook. Allowing incoming achievement alone 
to be the explanation for disparities in student outcomes in high school reinforces the 
idea that these differences are out of the control of educators, driven by what happened 
in the past, not by the current practices of adults in the high school. This is where the 
thoughtful disaggregation of data becomes so important. 
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However, if we compare the outcomes of students across racial/ethnic and gender 
groups whose prior grades and test scores positioned them to be at “high opportunity” 
for success in high school, the conversation changes. As shown in Figure 11, school 
teams may see that, when narrowing the focus to these high-achieving students, 

Black young men are far less likely than their similarly prepared peers to earn a 3.0 in 
freshman year. This view of the data highlights the disparities in outcomes between 
students of different races/ethnicities who share the same academic background. Now the 
focus is squarely on the practices of the adults in the building, and in this way, using 
different disaggregations can deepen a team’s understanding of students’ outcomes 

by sub-group and hold themselves accountable to a higher standard of improvement. 
When Freshman Success Teams combine these views—and bring a lens of equity to 
their analysis—they are nearly always able to observe that, even within a given student 


group, their practices are at the root of a group’s performance. 


| Figure 11: Outcomes for High Opportunity Freshmen Disaggregated by Race 
at Sample School 
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Even if disparities in student outcomes by race can be explained by looking at prior 
achievement, effective Freshman Success Teams use this fact as explanation, not an 
excuse. The conversation in these teams does not sound like, “Our Black males didnt 
have high grades in middle school so what can we do about it?” Rather it may sound like, 
“Our Black males may not have had great success in school in the past—so what are we, 

as educators, going to do to change that experience for them in high school?” As we have 
stated throughout these lessons learned, adhering to this focus requires a great deal of 
relational trust among team members. It also requires that educators commit themselves 
to the ongoing work of self-examination that is required to overcome implicit biases 


and proactively disrupt inequity. 


Data System Designers: 


Provide school-based educators with disaggregated data that 
allows them to answer difficult questions about inequity 


In order for educators to identify groups of students who are underachieving, and begin 
shifting how they see and treat these groups in school, they need very specific cuts of 
data. There are several essential questions that a data system must be able answer about 
each indicator, each leading to a different kind of data disaggregation: 


How are students doing across different demographic and learning 
differences (race/ethnicity, gender, diverse learners, ELs)? — Differences in 
outcomes or growth for different subgroups of students could suggest a bias on the part 
of some educators. It may also suggest a particular, common experience that a certain 
group of students are having inside the school. This may be indicative of a structural 
form of racism that exists in a school policy and/or implicit biases on the part of 


educators and in curriculum. 


How are students fairing across different levels of prior achievement? — 
Most high schools in Chicago serve students with a wide range of academic histories 
leading up to ninth grade. Disaggregating data on important outcomes by prior 
achievement helps schools understand whether or not their practice supports all 
students, regardless of prior performance. Differences across prior levels of attainment 
could suggest that the teaching and learning is directed toward a particularly well- 
prepared or particularly struggling group of students. As a result, a group of students 


with a specific level or prior attainment is possibly disengaged and/or disconnected. 


How are students doing in different grade levels, subject areas, and 
classrooms? — Differences in a particular subject or classroom can be most 
indicative of the effect of teacher practice on student learning. This suggests a practice 
used by an individual or group of teachers (grade level or department) is accelerating 


or hindering student learning. 
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Researchers and Analysts: 


Dig beneath descriptive differences in outcomes to uncover 
leverage points for educators 


For school practitioners to address inequities in educational outcomes, it is essential 
that the research and analysis behind the indicators must examine how these data 
apply across student subgroups. Oftentimes, analyses focus on estimating average 
effects on a total population or show that one subgroup outperforms another on a 
given outcome. These kinds of analyses don’t provide much insight as to why those 
gaps exist or whether the indicator itself might mean something different in its ability 
to predict future outcomes across student sub-groups. Researchers can do much more 
than shed light on the differences in outcomes by gender, race/ethnicity, EL status, or 
other subgroups. For example, in the years after the Freshman OnTrack indicator was 
identified as an important predictor of high school graduation, Consortium researchers 
returned to that foundational relationship to test whether the indicator held the same 
predictive power for students of two different subgroups: students who were or had 
ever been ELs, and students with diagnosed learning disabilities.”° Each analysis 
pointed to disparities in outcomes for these vulnerable student groups, but in each 
case, the on-track indicator held as predictive. This reinforced the message that, while 
supporting students to higher levels of course performance in ninth grade may prove to 
be a bigger challenge for some groups of students, it is still the right work to do. This is 
especially true if the goal is to increase high school graduation and college access for 

all subgroups of students. 


While supporting students to higher levels of course 
performance in ninth grade may prove to be a bigger 
challenge for some groups of students, it is still the 
right work to do. 


Similarly, Consortium research on the college-going outcomes for CPS students has 
consistently called out disparities in college enrollment by race as a challenge, but 

the analysis went further than simply identifying disparate outcomes. In the seminal 
research report, Potholes on the Road to College, Consortium researchers identified a 
significant gap between Latino students and their peers across race/ethnicity—a gap 
which only grew when controlling for students’ achievement level. This finding, if left 
uninvestigated, would have given very little guidance to practitioners on how to better 
support Latino students as they transition out of high school. In addition, this find 


might have had the unfortunate effect of reinforcing limiting beliefs that educators 
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may hold, consciously or unconsciously, about the extent to which Latino communities 
in Chicago value postsecondary education. Importantly, the analysis in Potholes also 
explained that educational aspirations were not meaningfully lower among Latino 
students, that Latino students came from communities with lower levels of educational 
attainment than their peers across the city, and most crucially, that Latino students 
appeared to be even more influenced by the college-going culture and structured 
supports for college access provided by their schools than other groups of students. 
Taken together, these findings removed some excuses that educators may have relied 

on to explain low college enrollment numbers. Rather, the analysis reinforced the 
message that schools have a role to play in shaping college access for students 


of all backgrounds.” 
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Interpretive Summary 


We have a long, frustrating history in American education of developing exciting 

ideas that fail to deliver on their promise because they cannot be implemented or 

scaled effectively. For example, a new curriculum that revolutionizes the practice of 

the teachers trained to use it, but that falls flat when a larger group of teachers applies 

it with less training. Positive outlier schools that produce impressively strong student 
outcomes but cannot reproduce their success when they replicate their model. Academic 
enrichment programs that work wonders in early-adopter schools but lose their purpose 
and impact as leaders push the programs citywide. The words of Charles Payne still ring 
dishearteningly true a decade after they were written: we have seen so much reform in 
our schools, and yet so little change.” 
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Veteran educators know this cycle well: A group of talented educators develops a smart, 
effective strategy to address a particular problem of practice. They try it in their own 
context, and it works. They document the results, recruit allies, refine the strategy, and 
try it out in two new schools. It works again. Ten schools later, the strategy turns into a 
packaged program that any school can pick up and use to train its teachers or support 
its students. In doing so, the strategy gets divorced from its original purpose, along with 
most of the nuance and all of the support that the original authors carried with them. 
To the educators who encounter it now, the once-great idea is just another PowerPoint 
they sit through in a rotating series of educational fads that have little relationship to 
their practice. Their school tries out the intervention for a year and then moves on to 
the next idea when the principal leaves or the funding is cut. When researchers look 
back at the limited impact of the once-great idea, they conclude that the promising 
practice failed to scale because educators on the ground lacked the capacity to 
implement it effectively. 


The most important lesson we have learned in 
developing this approach of practice-driven data in 
Chicago is that the data we provide to educators Is 
only going to be as helpful as the conversation that 
educators generate around it. 


But what if the intervention itself was more of an approach to increase educator 
capacity to drive improvement in their own school buildings? Not a program, 

or curriculum, or model per se, but rather an effort to develop educator capacity 

to recognize and solve their own problems of practice. Schools that have a deep 
understanding of their students’ data can identify, for example, that the high rates of 
class failure they see for male students is driven largely by a disproportionate suspension 
rate. So, they subsequently seek out strategies and supports to solve that problem 
specifically and bypass programs not suited to that particular problem of practice. 
Schools with high-functioning teams that monitor student outcomes alongside student 
interventions can make savvy decisions about when to stop using programs that are no 
longer working, and when to make deeper investments of their precious resources into 
programs that are working well. Schools that use data like this can insulate themselves 
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from “reform fatigue” and rely on their own data-driven professional judgment to make 
decisions about what strategies to use to improve. 


Building a school culture where educators use data in these ways is not impossible, 
but it is far from easy. Crucially, it is certainly not an automatic result of introducing 
data—even very actionable, high-quality data—into school life. Indeed, the very idea 


of being “data-driven” in our educational practice is on the cusp of being yet another 
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fad in education reform. The most important lesson we have learned in developing this 
approach of practice-driven data in Chicago is that the data we provide to educators is 


only going to be as helpful as the conversation that educators generate around it. 


The practice-driven data approach we have described in this paper is a marathon, 

not a sprint. It adheres to a set of principles: a deep respect for the professionalism 

of educators, a recognition of context and autonomy, and an acceptance that 
implementation is often as messy and ambiguous as the system itself. Practice-driven 
data has no playbook, no singular practice, no such thing, really, as fidelity. However, 
there is much that education leaders, policymakers, and thought leaders can do to 
support this approach. They can generate data that sets the right tone. They can 
explicitly invest in the time, training, and support that schools need to prepare to have 
hard conversations. They can prioritize a small number of research-based indicators to 
keep educators focused on what matters most. They can provide research and analysis 
that helps educators interact with the evidence directly and build a sense of ownership 
over the problem they hope to solve. They can provide data that reaches the right 
people at the right level of detail at the right time. And we all can commit to using data 
to disrupt inequity in our schools. 


We have tried to capture these lessons because we have seen, first hand, the promise 

in their application at scale in Chicago. In high schools across the city, you will find 
principals changing school culture and organizing their schools for improvement. You 
will see assistant principals and teacher leaders leading permanent, stable teams of 
educators to relentlessly tackle challenges like instructional improvement and getting 
freshmen on track to graduate. You will see counselors organizing every adult in the 
school to strategically support seniors’ college application processes. You will see them 
all using data to guide their work. These practices are by no means universal, and the 
challenges these educators face are still significant. But even given the depth of the 
challenge in Chicago, the limited resources educators have to draw on, and the frequent 
changes in leadership at all levels of the school system, every year in our city, more 
ninth-graders are on-track to graduate, and more seniors enroll in college. We hope 

the approach we describe in this paper will remind our Chicago community that the 
educators of Chicago are capable of driving remarkable change in a large and complex 
system as we all seek to catalyze another decade of improvement in the lives of Chicago's 
young people. We further hope that these lessons will help educators in districts and 
cities across the country spur even greater change for all of their students. 
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DuFour & Eaker (1998). 
Payne (2008). 
Nagaoka, Seeskin, & Coca (2017). 


For some examples of protocols and other tools 
used to guide teams’ conversations, please see the 
NCS Freshman On-Track Toolkit (https://ncs. 
uchicago.edu/freshman-on-track-toolkit) and the 
NCS Postsecondary Success Toolkit (https://ncs. 


uchicago.edu/postsecondary-success-toolkit)/. 


Allensworth, Gwynne, Moore, & de la Torre 
(2014). 


More information on how these “risk and 
opportunity” categories were created is available 

on the NCS Freshman On-Track Toolkit here: 
https://ncs.uchicago.edu/tool-set/ongoing-research- 


uchicago-consortium. 
Allensworth & Easton (2005). 
Allensworth & Easton (2007). 
Allensworth & Easton (2007). 
Roderick (2012). 


Roderick et al. (2008). 
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Roderick et al. (2008). 


For more information about college match, 

see Roderick et al. (2008) and/or https://ncs. 
uchicago.edu/sites/ncs.uchicago.edu/files/ 
uploads/tools/NCS_PS_Toolkit_URAD_Set_C_ 
CalculatingCollegeAccess. pdf. 


For more information about how this roster 

is generated and used in schools, see https:// 
ncs.uchicago.edu/sites/ncs.uchicago.edu/files/ 
uploads/tools/NCS_PS_Toolkit_URAD_Set_C_ 
CalculatingApplicationMatch.pdf. 


Allensworth & Easton (2007). 
Allensworth et al. (2014). 
Ladson-Billings (2006). 


For more information on middle grades predictors 
of high school performance see Allensworth et 

al. (2014). For more information about “risk and 
opportunity” groups, see https://ncs.uchicago.edu/ 
tool-set/ongoing-research-uchicago-consortium. 


Gwynne, Lesnick, Hart, & Allensworth (2009); 
Gwynne, Pareja, Ehrlich, & Allensworth (2012). 
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the 
To&Through 
project 


UCHICAGOConsortium 


on School Research 


The University of Chicago Network for College Success 


The mission of the Network for College Success (NCS) is to cultivate postsecondary 
readiness and success for all students by translating research into practice and 
supporting high school leaders to organize their schools for improvement and 


innovation. 


Since 2006, NCS has partnered with Chicago Public Schools to dramatically increase 
high school graduation, college enrollment, and college graduation rates for all 
students. Housed in the School of Social Service Administration at the University of 
Chicago, NCS helps educators across the city and nationally respond to emerging 
research, analyze in-time student data, engage in joint problem-solving, and share 


effective practices to improve student outcomes. 


The To&Through Project 


The To&Through Project is a partnership between the University of Chicago’s 
Urban Education Institute and the Network for College Success. Our mission is 
to use research, data, and professional learning to help more students get to and 


through high school and college: 


¢ Research that illuminates what matters most for students’ high school and college 


success 
e Data that guides efforts to improve students’ attainment of key milestones 


¢ Professional Learning that helps translate research and data into improved 


practice. 


In collaboration with educators, policymakers, and communities, the To&Through 
Project aims to significantly increase the percentage of the Chicago Public Schools 
freshmen who graduate from high school and go on to earn a college degree, and to 
share the learning from Chicago with education stakeholders across the country. 


The University of Chicago Consortium on School Research 


The University of Chicago Consortium on School Research (UChicago Consortium) 
conducts research of high technical quality that can inform and assess policy and 
practice in the Chicago Public Schools. We seek to expand communication among 
researchers, policymakers, and practitioners as we support the search for solutions 

to the problems of school reform. The UChicago Consortium encourages the use 

of research in policy action and improvement of practice, but does not argue for 
particular policies or programs. Rather, we help to build capacity for school reform 
by identifying what matters for student success and school improvement, creating 
critical indicators to chart progress, and conducting theory-driven evaluation to 


identify how programs and policies are working. 
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PRACTICE-DRIVEN DATA 


NETWORK : 
& FOR COLLEGE roéThrough UCHICAGOConsortium 
> SUCCESS project on School Research 


We write this paper as three entities that are a part of the University of Chicago and 
partners to the Chicago Public Schools: the Consortium on School Research, the 
To&Through Project, and the Network for College Success. This paper is designed 
to share the lessons we have learned about how, when, why, and under what 
conditions we have seen the use of data support real, sustainable, and remarkable 
improvement in the outcomes of Chicago’s youth. Our work as partners to the 
school district has implications for advocates, policymakers, district and school 
leaders, and school support organizations, both locally and nationally, who wish to 
take the approach to data that we have seen work so well in Chicago. We call this 
approach practice-driven cata. 
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